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Exploring EFL learners’ lexical 
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Abstract. With massive utilization of Automated Writing Evaluation (AWE) 
tools, it is feasible to detect English as a Foreign Language (EFL) learners’ lexical 
application so as to improve their writing quality. This study aims to explore 
Chinese EFL learners’ lexical application to see if AWE-based writing can bring 
about positive effects of lexicon on linguistics features, especially in word count, 
word frequency and word density in their writing texts across a series of tracked 
revisions in University English classrooms and beyond. The findings from both 
classroom observations and learners’ responses to a post-test questionnaire survey 
reveal that they showed very positive perceptions of AWE-based writing in which 
they were greatly encouraged to make modifications in each writing task. It shows 
from data analysis of the writing texts that learners tended to replace simple 
lexical terms with more academic and concrete ones in revisions. The results 
signify implications for EFL learning of lexicon and writing, and also for AWE 
developers. However, it implies from the analytic features of lexicon that learners’ 
lexical modification was limited. This perhaps suggests for further research on 
learners’ lexical modification in AWE-based writing context to see if there is any 
difference across genders and different levels of achievers. 
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1. Introduction 

Learners’ writing proficiency partly depends on their lexical ability in terms of 
writing improvement, as writing consists of ‘grammaticalized lexis’ (Lewis, 1993). 
Lexicon, not merely a list of words, is conceived as “a set of lexical resources, 
including the morphemes of the language, plus the processes available in the 
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language for constructing words from those resources” (Trask, 1 999, p. 1 1 1). In this 
study, lexical application ability refers to learners’ effective use of lexical items, 
including words and phrases, frequent collocations, institutionalized utterances, 
sentence frames and heads (Lewis, 1993). Previous research has proved a positive 
relationship of lexicon on language learners’ writing quality in terms of word 
range, word diversity, word frequency, and polysemy (Engber, 1995; Hu, 2015; 
Qin & Wen, 2007; Laufer & Nation, 1995). 

Recently, massive application of AWE tools has been expanded to language 
teaching and learning purposes, which has provided great potential possibilities 
for EFL learners to proceed with their writing tasks at their own rate with real- 
time feedback. Much research is reportedly promising and positive, where the 
utilization of AWE tools can be realized in terms of time efficiency, reduction of 
teachers’ workloads, increased learner autonomy (Sherimis & Burstein, 2013) and 
improved writing skills (Scharber & Dexter, 2008). This study aims to explore 
EFL learners’ lexical application to see if AWE-based writing has positive effects 
of lexicon on linguistic features based on their writing tasks on a web-based 
AWE tool, the Pigai system (http://www.pigai.org), which makes it accessible to 
keep precise track of any step in the creation of writing with real-time feedback 
in terms of vocabulary use, grammar, cohesion, and content relevancy. 


2. Method 


2.1. Research questions 

This study is driven by the following questions: 

• Is the AWE tool, the Pigai system, helpful for students to improve their 
writing skills, especially in terms of lexical application? 

• If yes, to what extent does the AWE-based writing affect students’ lexical 
application? If not, what are the causes? 

2.2. Teaching and research design 

The study was carried out in one-week intervals from September 2015 to 
January 2016 and it involves 26 second year non-English majors in Beijing 
University of Posts and Telecommunications’ English Audio-Video Speaking 
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Course (EAVSC), with a pre- and post-test design coupled with follow-up 
questionnaires in a digital lab. All of the students had passed CET-4 3 before taking 
the course but were still required to take a test of the same format of content-based 
integrated listening-speaking tasks through a self-developed web-based English 
language skills training system at the beginning and the end of the semester. 
During the whole semester, classes were conducted using the following pattern: 
students began every class by watching a video clip; next, engaged in synchronous 
computer-mediated group and pair discussions on the given topics; then proceeded 
with a ten minute online writing task on the same topic using the Pigai system; 
finally, performed a one minute recorded personal statement task on the same topic. 
The AWE-based ten minute writing task in class with an assigned topic each week 
served as an embedded task on a regular basis and students were allowed to revise 
their writing after class before a deadline set by the instructor. 

Figure 1. A screenshot of students’ writing submissions on the Pigai system 
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2.3. Data collection 

The questionnaires and interviews following pre- and post-tests were conducted 
and collected through the adopted system, which would provide insights into 
students’ perceptions of the Pigai system, particularly their feedback in terms of 
vocabulary, grammar, content and cohesion. All of the students submitted their 
pre-test questionnaire, but one missed the post-test questionnaire. All the students’ 
writing texts were collected and each modification or reconstruction was tracked 
through the Pigai system. 


3. CET is the abbreviated form of ‘College English Test’. The national College English Test Band Four — CET-4 in China aims 
to evaluate non-English majors’ comprehensive language proficiency. Apart from CET-4, there is also CET-6, which is widely 
used to evaluate above-average college students’ English language proficiency. 
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2.4. Data analysis 

There were 1 1 writing tasks during the whole semester which covered a variety 
of social issue topics, including job hunting, the relationship between tourism 
development and environmental protection, e-games, etc. 

In order to explore the students’ lexical application and the effectiveness of lexical 
modification in AWE -based writing, the multiple writing scores and lexical 
features of the three writing tasks (the first, sixth, and final ones) were selected as 
the research samples in the current study. The scores data were processed using 
SPSS 22.0 to see if there is a significant difference in the first and final draft in 
each of the three writing tasks. 

To investigate students’ lexical application across multiple drafts and their 
lexical modification in AWE-based writing, the Coh-metrix tool (Graesser, 
McNamara, Louwerse, & Cai, 2004) was employed as a complementary tool 
for analytic features on texts. 

The web-based Coh-metrix, developed by the University of Memphis, automates 
over 200 measures of language, discourse and cohesion through extracting 
information from texts. Change in lexicon use can be seen from the measured 
scores of texts calculated with Coh-metrix, including word count, word length, 
word frequency, and lexical richness across multiple writing results. 


3. Results and discussion 

3.1 . Analysis of AWE-based writing scores 

The average number of revision times of the 11 writing tasks is 5.12 to 13.42 (in 
the seventh task). Although it drops to 6.81 in the final task after 12.69 for the 
tenth task (probably because it was close to the final exam session), as shown in 
Table 1 , students in the mid and final writing tasks still conducted more revisions 
than they did in the first task. 

The results of the students’ writing scores basically fit the normal distribution and 
paired sample t-tests revealed a significant difference in scores between the first 
and the final drafts for all of the three writing tasks in the study (tfirst task= -6.06, 
p <0.05; tsixth task= -5.50, p<0. 05; tfinal task= -7.34,/><0.05). 
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Table 1. Paired sample t-test of scores for students’ drafts 



N 

Average 
times of 
revision 
drafts 

Mean Scores 


First draft 

Final draft 

MD 

t 


M 

SD 

M 

SD 


First writing task 

26 

5.12 

70.06 

9.78 

75.90 

8.11 

-5.84 

-6.06* 

Sixth writing task 

26 

7.31 

69.04 

12.74 

78.50 

8.22 

-9.46 

-5.50* 

Final writing task 

26 

6.81 

63.83 

7.28 

72.25 

7.45 

-8.42 

-7.34* 


Analytic features of students’ perceptions 

With respect to how students perceive the Pigai system in affecting their writing 
quality, from the questionnaire feedback, 84% students considered that for each 
revision they valued the score provided by the system, and only 20% of them thought 
they would cease modification if they get a lower score after their modification. 
68% of them showed gratitude to the real-time and multiple comments which 
helped them realize their weaknesses and pushed them to rewrite or revise. 

3.2. Analytic features of lexicon 

As is shown in Table 2, the paired sample t-test results revealed that there was no 
significant difference between the first and the final draft both in average word count 
(t=-1.89,/>>0.05) and word frequency (t=0.67,p>0.05). The word frequency count 
was based on CELEX, the database from the Dutch Centre for Lexical Information 
(Baayen, Piepenbrock, & Gvdikers, 1995). For the average word length, students 
chose longer words in their final drafts than those in their first drafts (t= -4.44, 
p<0.05), indicating that they possibly used more academic or formal words in the 
process of revision. 


Table 2. Analytic features of the final writing task 


Linguistic features 

N 

Drafts for the final writing task 


First draft 

Final draft 

MD 

t 

Average word count 

26 

138.80 

159.35 

-20.54 

-1.89 

Average word length 

26 

4.14 

4.26 

-0.12 

-4.44* 

Word frequency 

26 

3.08 

3.07 

0.01 

0.67 

Type-token ratio 

26 

0.62 

0.60 

0.02 

1.49 

Connective incidence 

26 

103.63 

1.09.39 

-5.75 

0.013* 


Note: *p<0.05 
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Type-token ratio is very important to measure the lexicon richness, for a low ratio 
usually indicates that words are repeated many times. Table 2 shows that type- 
token ratios of the first and the final draft had no significant difference (^=1 .49, 
p>0.05). However, the results revealed a significant difference in connective 
incidence (t=0.013, p<0.05). The higher the incidence of connectives, the more 
it is assumed that students applied connective words and phrases in their final 
drafts. 


4. Conclusions 

The findings from the research showed that students appreciated and valued the 
real-time feedback provided by the Pigai system and they were highly motivated 
and encouraged to make modifications for each writing task. They tended to 
replace simple lexical items with more academic and concrete ones. Nevertheless, 
the analytic features of lexicon implied that their writing improvement was 
limited, and this perhaps requires for a further study on textual analysis in depth 
so as to better explore their modification tracked by the Pigai system to see 
if there is any difference across genders and among different English levels of 
students. 
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